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Abstract 

Derrida's model can be used for construction and transmission of error correcting 
codes, in an optimal way. We use the extreme statistics method to derive the optimal 
signal to noise ratio. 

0^ 

^ Introduction 

I ^ A given message can be coded into a sequence of N bits a = {cr, ,---,(7^},(T, = ±1 . 

^ Once transmitted through a communication channel, we may receive a different 

^ sequence say <7\ depending on the statistical properties of the channel. Clearly 

^ decoding may result in obtaining a corrupted message in place of the original message. 

However if we had allowed for some redundancy in the original coding, allowing for 
O the errors resulting from the communication channel one could stiU recover the original 

message, uncorrupted. More precisely, instead of the original sequence we transmit: 

0\ 

§ K=c^..... cy...- ' k = \,-,M (1) 



I We have thus replaced the original sequence with a longer sequence {M > N), the 

price we have paid for immunity from information corruption, is that we transmit a 
O longer sequence. The connectivity matrbc C^*' • • • . which has elements zero, or one, is 

\^ Al kp 

N 

in fact the "rule" for constructing the error correcting code. The ratio R = — , called 

M 

the rate of the code, counts its redundancy. The tasks which face us are, first determine 
the connectivity matrix in such a way that R is maximised, second decode the 
transmitted message in an efficient manner in order to recover the original message. 
Shannon's theorem sets a limit on the rate, in terms of the channel capacity. The 
channel capacity is the maximum information per unit time which can be transmitted 
through the channel [1]. 

In general, when a signal a , is sent through a channel, the output wiU be a real number 
X . Let2(M|cr)du be the probability of observing the output between wand u + du 

given an input a . A channel is called memoryless if the noise is independent of the bit 
transmitted: 

Q(x\a)=nQ(x]a,) (2) 



Shannon shows that for a memoryless channel and gaussion noise the channel capacity 
is given by: 



1 r (3) 

2£n2 0)^ 
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V 

Where — is the signal to noise ratio. According to Shannon the rate R is always less 
0) 

than or equal to C . However Shannon shows that ideal codes which saturate the bound 
exist. Spin glass theory provides us with a way of constructing such codes. 
Sourlas [2] has pointed out that the p-spin hamiltonian of Derrida [3] provides a good 
example of error free coding. The original message sequence a is the ground state of a 
p-spin hamiltonian. Thus the task of decoding is equivalent to finding the ground state 
of p-spin hamiltonian. The hamiltonian is constructed in such a way that it operates in 
the ferromagnetic phase, thus there exist a unique global ground state, in the absence 
of the channel noise. In the presence of channel noise the bonds take on random 
values, thus the existance of a global ground state becomes questionable. However, the 
parameters are tuned in such a way that the system remains in the ferromagnetic phase, 
therefore despite the noise there is stiU a chance of recovering the original message. 
More precisely, the input is the sequence defined by equation (1), the output is a 
sequence ofM, real numbers/^"', which are random and obey the distribution 

Q{jk"' / -^k] satisfying the constraint set out in equation (2). 

Now the probability that any sequence cr is the original message conditional on the 
output J'i;"',p(c7\r"') is given by [4]; 

lnp{a\j""')=const-H{a)+Y,cl^^^-,^ B,a,^^ -a,^ (4) 

where H{a) gives the bias in the original code, this may be constant. The coefficients 
are given by: 



1, Q{jrn) (5) 

= — In — 7 r 

2 e(/r/-i) 



Equation (4) is indeed the defining relation for Derrida' s p-spin model. The most 
probable code is the ground state of the hamiltonian defined by (4). As a concrete case 
let us consider the fuUy connected version where all c'*'are set equal to 1. Let all 
codes be equally probable, h(<t) = 0, and let the channel be gaussian. Then, we have the 
following hamiltonian: 



(6) 



Where / are output values with a normal distribution: 



(7) 



P(/„J~exp. 



J- - J..N 



N 



p-i 
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Where we have assumed that i « p « a^. Here the original code is assumed to be the 
trivial signal where aU cr =1. This is not a resticting assumption since, hamiltonian (6) 
is gauge invariant under the transformation a^^a.s^ ,s^=+\- Thus aU codes are 
equivalent. 

The procedure for decoding is equivalent to fiding the ground state of the hamiltonian 
(6). Of course the rate is very poor: 



R = 



N 



p-i 



(8) 



As is well known spin glass hamiltonians such as Eq. 6 can have many minima, thus the 
task of decoding is formidable, unless the system operates in its ferromagnetic phase. 
In this case there should exist a unique ground state. The phase diagram of Derrida's 
model in preseme of a ferromagnetic interaction was derived in [5, 6]. Indicating that 
there does indeed exist a region in phase space, even for finite N, where a unique 
ferromagnetic phase exists. The critical value of signal to noise ratio is: 



0) 



> (2l«2)2 



(9) 



In this paper we aim to derive this result using extreme value statistics. A much more 
powerful tool which readity yields the result. 



Extreme Value Statistics 



Consider M independent identically distributed random variables £. ,j =i,l ,m, such 
that the probability distribution p[e) decays faster than any power in infinity. What is 
the distribution of E' the minimum value of £. , for large M ? 

This is clearly a statistics which is suited to the study of the lowest energy levels in 
disordered systems [7]. The cumulative probability p^[E) is defined as 

E 

P,{E)= jdxpix) 

The distributions of E' is 

P^{E^) = MP{E'){l-py-=-^[p^{Ef (11) 

For large M , the minimum E' will be negative and large. Thus p^[e') will have a peak 
near e^ ; determined by the equation mp^{e^) = i . 



(10) 



Now for our model hamiltonian M = 2" is indeed very large. Thus we can find the 
distribution of the ground state energy using the method above and the fact that for 
large P, Derrida's model is approximated by the random energy model [3]: 



This immediately leads to the value: 

E^=-N{2en2)\ (13) 

Now if a ferromagnetic term is present the distribution of the energies wiR be slightly 
different. The model is stiU defined by (1 1) except for one energy level with a different 
distribution due to the presence of : 

_(E+NJ^ (14) 

p{Ea)~e ^""^ ^ ' 

We thus need to modify equation (10) to take account of the fact that one energy level 
behaves differently: 

I^E-)-J^\K(Er'KiE-)] (15) 

Where is defined using the normal distribution with a mean at -A//^ rather than zero. 
We now find that the distribution p{e*^ still has one peak at provided that 



— < 



{2ln2)\ (16) 



Where as beyond this value, the distribution wiU have two peaks, with the lower peak 
at -w , that is the ferromagnetic phase. Therefore for efficient decoding one must 

have l^>{2in2)\- 

w 

Let us next consider Derrida's model, with a binary distribution for the j.^. , namely 



2 



I -can 



2 



J 



where a = -\[N^p\. This is more appropriate to a digital channel, again we consider the 
fully connected case. We can show that the distribution for energy factorizes in the 
limit of large p , and zero magnetization: 

/(£,-£„)=n/(£„) 

a 

where 



f{E)={^e"^x{CosaA) 



where x = 



. Expanding we have alternatively: 











r=0 


J. 



Which is approximated by a normal distribution of zero mean and variance J^N . We 
can thus repeat last section's argument to get the critical value 

= -N{2ln2)i . 



and 



=(2 log 2)2. 

Concluding Remarks 

We have observed that extreme statistics offers a very powerful tool for dealing with 
systems with random energy levels. In particular extreme statistics immediately gives 
the threshold values for operation of Derrida's model as an error correcting code. The 
calculation equally well works in gaussian and digital channels. 

Fully connected Derrida's model can saturate Shannon's limit, but this is stictty true 
only in the limit of large N , or vanishing capacity which is not practical. 
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